Tools of the Trade

Session 1

Session Outline

Understanding Big Data

The Five V's
Developing The Future
How It Works
Using Big Data in your Role
Advantages and Disadvantages
Big Data Products
Data Platforms
Platforms vs Coding Yourself
Setting Up R
Recap

Learning Objectives

Understand fundamental concepts of Big Data
Justify the use (or lack of) Big Data technologies in your analysis
Critically evaluate the differences in using a Data Platform and Coding it Yourself

Understanding Big Data

What is Big Data?

Big Data is data whose scale, distribution, diversity and or timeliness require the use of new technical architectures and analytics to enable insights that unlock new sources of business value.

McKinsey & Co.; Big Data: The Next Frontier for Innovation, Competition, and Productivity

Generates 500 million Tweets every day

Source: Internet Live Stats

Generates 1 TeraByte of trading data every day

Source: New York Stock Exchange

Generates 3 PetaBytes of user data every day

Source: Facebook

79% of enterprise executives agree that companies that do not embrace Big Data will lose their competitive position and face extinction

Source: Accenture

The Five V's

Volume

Velocity

Variety

Veracity

Value

Volume

Refers to the size of the data

Whether a dataset is considered 'big' or not is dependent on the volume

Data is considered Big when it is too voluminous for traditional data storage methods

Big Data platforms provide a way to store incredibly large amounts of data as well the ability to process it efficiently

Velocity

Refers to the speed of data generation

For example, how fast data flows in from sources such as application logs, networks and social media sites

Typically the flow will be massive and continuous

Velocity

Increasingly companies are looking to stream analytics and data so it available at the right time to make appropriate business decisions

Big Data platforms offer solutions to dealing with fast flowing data in terms of storage and processing

Variety

Refers to the heterogeniety of the data sources

In other words, how different the data sources are (unstructured vs structured etc)

With large flows of data from several of sources it is possible that there will be a variety of data types and structures

Variety

Big Data platforms can handle a variety of sources, structures and data types

Different types of data require different strategies which you will need to consider as part of your analysis

Veracity

Refers to the quality of the data

Checking the veracity of data sources is always challenging- never trust data as given

Enquire about the data preparation- has it been normalised already? Has it been subject to statistical manipulation (i.e. outliers removed)?

Where possible you should always ask to look at raw data

This refers to the quality of the data. What pitfalls can arise due to poor quality data? These issues are much worse with Big Data as there is so much of it, and any errors can lead to catastrophic failures which can be both expensive and time consuming. You should always have checks then for your incoming data- has it been normalised? Has it been cleaned? Have the structures been checked and processes put in place to deal with them? Never assume data is clean! As Big Data often comes from a disparate variety of sources, this can be challenging- especially when you also have to consider issues around ownership, GDPR and company data policies. Considering how expensive it can be to use Big Data, you do not want to find your analysis has been ruined due to issues around data quality- it could be both a frustrating and expensive mistake.

Veracity

Determining veracity can be challenging to an organisations policies as well as externally determining authorship/ownership

Value

Refers to the business value of the data

Having access to vast amounts of fast flowing data is useless unless it can leveraged into something of value

How does this data add value to your business? What is the ROI?

Activity

Think about some of the data you use regularly, how does it stack up against the 5 V's?
Share your thoughts with your group and be prepared to feedback to the rest of the class

Developing the Future

How has Big Data affected medical science?

Genetic sequencing and human genome mapping provide a detailed understanding of genetic makeup and lineage.

The healthcare industry is looking toward these advances to help predict which illnesses a person is likely to get in their lifetime.
While data has grown, the cost to perform this work has fallen dramatically.
The cost to sequence one human genome fell from $100m in 2001 to $10k in 2011.
Today it costs around $1k.
A 2017 report showed it was possible to train a model to detect tumour in breast cancer patients that either matched or exceeded the performance of a pathologist who had unlimited time to examine the slides.

Natural Language Processing

A powerful tool that allows the analysis of human languages, e.g. sentiment analysis and key word idenfitication. Companies like Google and Amazon make use of NLP and other technologies to give us a virtual assisstant experience.

Activity

In groups, discuss and sketch out the process you think Alexa would follow to provide a user with a recommendation

Your words are recorded
The recording is sent to Amazon's server to be analysed (due to interpreting sounds taking up alot of computational power
Important words are identified and corresponding functions are carried out (e.g. 'basketball' would open a sports app)
The server then sends the information back to your device. If Alexa needs to say something it will follow a reverse process (identify the words it wants to give and convert them to the correct sounds)

How it Works

Divide and Conquer

Most Big Data softwares use an approach called 'Divide and Conquer' where the data will be split into smaller 'chunks' and processed simultaneously across different nodes and the results combined. This process happens over several servers so if one fails the analysis can continue and the remaining can pick up the slack.

Has anyone ever tried to run analysis on a dataset that runs into millions of rows? How long did it take? When using Excel or a Jupyter Notebook the processing is being done by your computer. There are very powerful personal computers you can buy, but these are often optimised for computer games. Most of the computers we use are only powerful enough to handle a few Gigabytes which severly limits how quickly (and how much) data can be processed.
This is where Big Data technologies come to the rescue. Instead of running your algorithm on your computer, you send the data to a remote server (run by a Big Data company) where your algorithm/script will run. The server uses an approach called 'divide and conquer' where it will share the data across several nodes and process them seperately before combining the results and sending them back. When you consider how powerful the processors these servers have you can see how much faster your analysis will be completed when using Big Data technology. This is especially true when the data is in large volume and fast moving, with Big Data servers able to store huge quantities and even have capabilities to deal with different data types and structures. Another advantage of spreading the data across different nodes is that if one fails for whatever reason, the others can pick up the slack. Compare this to running analysis on your computer- if it loses power midway through a large piece of analysis you will lose everything.

Which is Faster?

One 2TB Harddrive

Ten 128GB USB drives

Ask the apprentices for their answers (can use annotate or chat function) before revealing solution. Note that the second method is 10 times faster. This shows how much quicker dividing your processing up can be when using Big Data technologies. However, sending your data to a Big Data server can be expensive as you are effectively 'renting' someone else's processors. What you need to weigh up is the cost of using the technology against the added value of processing larger amounts of data more quickly and efficiently (again when you consider the capabilities of dealing with different data types). Something else to consider is accuracy- you do not want to make a mistake. Sending some data to the cloud to find that you had made a syntax error or some of the data was corrupt (or not cleaned properly) will not only be frustrating but potentially expensive to your employer as well.

Using Big Data in your Role

Activity

In groups, discuss how you think Big Data technologies can be used to enhance your role

Advantages and Disadvantages

Advantages

Ability to access and process large volumes of data quickly
Analysis run on larger datasets will be more reliable, representative and accurate as taking samples from data will not be necessary
Faster analysis leads to a reduction in human utilisation, meaning more projects can be undertaken or better resourcing of analysts
Projects finished in shorter time frames can lead to increased reputation amongst customers and products being brought to market

Disdvantages

It is expensive
Big Data technologies only provide a small amount of free processing before charging and costs can rack up quickly
Therefore you do not have time to play around or make mistakes- time is money
Big Data still requires a degree of competency from the user, it is not something that can just be picked up by anybody
The nature of Big Data means you are exporting data externally, potentially bringing in cybersecurity risks
It can be challenging to integrate Big Data output into your system

Activity

In breakout rooms read and discuss this article .

Do you agree with what has been written?
How do you think Big Data will personally affect your role?
Can you find other articles which support or argue against the use of Big Data technologies?

Big Data Products

Which products are you aware of?

Ask the group if they are aware (or have used) any technologies already. If they have, ask them to explain how they used it and what the benefit was. Run through some of the examples. Hadoop was one of the first developed (named after the developer's daughter's toy elephant). Licensed by Apache, this software came up with the MapReduce (divide and conquer) method which revolutionised data analysis as we know it.
Also licensed by Apache is Spark, developed by the University of California, Berkeley this technology excels at parallel processing and dealing with faults.
AWS (Amazon Web Services) is a cloud based server owned by Amazon which allow users to store and process data, host scripts (such as plotly dashboards) and various analytics. Fun fact: AWS makes as much money for Amazon as the rest of the business combined.
A recent technology is Microsoft Azure which allows for cloud based computing which integrates well with other Microsoft softwares such as PowerBI
BigQuery is owned by Google and is effective at running SQL queries on a large scale.
When it comes to Big Data technologies, it is up to you (and your employer) which one to use. Whatever the decision, it is important in your portfolio that you justify why you used that technology (or didn't if it was available).

Data Platforms

Visualisation Software

Data Management

Statistics Platforms

What do you use?

Big Data is not the only technology that you have available to use. Your employers have likely licensed several tools you can use to help you in your analysis. In this section we will discuss a few and consider the advantages they have. The aim is again for you to think about the tools you use and justify their use in your portfolio.
First we will consider visualisation suites such as Tableau or PowerBI. Built upon pre exisiting code, these allow you to produce impressive visualisations quickly and without a technical knowledge of coding. Allowing for the incorporation of company branding, it is simple to build dashboards and powerful visuals to support your analysis and presentations. Tableau has capabilities for working with live data as well as analytics tools (PowerBI has these as well) which can help you leverage insights without having to code. Drawbacks include not having complete control over the building process as well as being limited to whatever has been built in. Can anyone think of other pros and cons of using visualisation softwares?
Most of us are aware of Microsoft Excel as a tool for managing data. With an intuitive interface and the ability to manipulate data down to the cell level it is often the go to choice for analysing data (especially EDA and calculating aggregates). Excel does have data limits (1 million rows) and searching/filtering can be tedious when compared to Python. Excel does integrate with Jupyter Notebook and Python allowing for a best of both worlds approach.
In DS2 we looked at hypothesis testing in Python and in the next session we will start looking into R as a language for statistcal analysis. Both Python and R require a technical coding knowledge, meaning many employers and academic institutions look to software like SPSS (owned by IBM). SPSS uses a tabular approch not too dissimilar to Excel and has a range of inbuilt statistical analysese with simply prensented outputs. However, it is expensive to license and has been around since 1968- and it shows. It is not reactive to the latest developments and again you are limited to what you have built in.
Ask apprentices to describe the tools they use, like Knime, Alteryx, Qlikview, Jira, etc. How do they use them? What are the advantages and disadvantages? Encourage them to think in terms of business context. Stress again that they need to justify these reasons in their portfolio.

Platforms vs Coding Yourself

Advantages

Platforms

Easy to use and do not require technical knowledge or skills
More agile as 'drag and drop' options allow for quicker building of functions/models/visualisations
Greater agility leads to lower costs as less time is spent on development
Relatively simple to edit what you have built
Often tech support available
Accessible to stakeholders who do not have technical coding background

Coding

You have complete control over the whole process and can customise colour scheme and functionality in almost any way imaginable
Data entry can be faster and more accurate (particularly when automated)
Common coding langauges (R, Python, etc) are free to install and use
You own the source code
Most languages have active communities where you can ask for help

Disadvantages

Platforms

Can be a steep learning curve and training will be required for each different software you use
While platforms often have a wide variety of functions, they are still limited to whatever was built in by the developer
Security and reliability issues around you not having complete control over your code/data

Coding

Learning to code is a daunting and long process
Coding is constantly evolving and it is up to you to keep up with trends- which you may not have time to do
Complicated codes take a long time to produce and require constant error checking and validation

Disadvantages (Cont)

Platforms

If you want to change product it can be difficult to migrate your models/visualisations
Licenses can be expensive
You do not own the source code

Coding

Complicated codes take a long time to produce and require constant error checking and validation
Easy to make mistakes but can be difficult to find the error
Making small changes to large scripts not always simple and can lead to side effects
Code is harder to understand for non-technical colleagues

Activity

Mr Jones works in a data analytics department and has been given a project to complete. He must design a dashboard that displays daily KPIs for his stakeholders. In teams you will be each be assigned a different product (Tableau, Python, etc) and will come up with arguments to convince Mr Jones to use your product. Each team will then be given 1 minute to pitch their product.

Setting up R

R set up and testing

Recap

Learning Objectives

Understand fundamental concepts of Big Data
Justify the use (or lack of) Big Data technologies in your analysis
Critically evaluate the differences in using a Data Platform and Coding it Yourself

Additional Resources

Big Query Webinar

Complete Session Attendance Log and Update Your OTJ